Use async processing to avoid database timeouts #437
Conversation
blootsvoets
requested review from
yatharthranjan,
mpgxvii and
peyman-mohtashami
| @@ -55,7 +59,9 @@ public MessageStateEventListener(ObjectMapper objectMapper, | |||
| * | |||
| * @param event the event to respond to | |||
| */ | |||
| @EventListener(value = NotificationStateEventDto.class) | |||
| @Transactional(propagation = Propagation.REQUIRES_NEW) | |||
There was a problem hiding this comment.
we have had transaction lock issues with postgres (aws rds) when using Propagation.REQUIRES_NEW previously, @mpgxvii can you elaborate the issues we faced previously.
There was a problem hiding this comment.
Indeed, using TransactionalEventListener + Async was essential, because it only opens a new transaction once the old one is committed. In earlier tests with other permutations, the transactions would overlap and block the HikariCP thread pool.
There was a problem hiding this comment.
Ah ok I see, sounds good. Yes we had to remove the REQUIRES_NEW propagation previously because it was resulting in "Too many connections" issue which was blocking other transactions and causing inconsistent states.
There was a problem hiding this comment.
@mpgxvii perhaps we can test this with our stage cluster?
In the previous handling of events, event were handled immediately during the transaction, in the same thread. This caused a lot of delays, blocking the database threads. The EventListener annotations are replaced by TransactionalEventListener to ensure that the previous transaction has been committed before processing the event. It explicitly creates a new transaction to avoid overlapping transactions. Finally, it enables async processing to avoid the existing transaction threads being blocked.
blootsvoets
force-pushed
the
limitTransactionTime
branch
from
a030664
to
c73d2d9
Compare
In the previous handling of events, event were handled immediately during the transaction, in the same thread. This caused a lot of delays, blocking the database threads. The
EventListenerannotations are replaced byTransactionalEventListenerannotations to ensure that the previous transaction has been committed before processing the event. It explicitly creates a new transaction to avoid overlapping transactions. Finally, theAsyncannotation enables async processing to avoid the existing transaction threads being blocked.This fixes an issue that caused timeouts if the fcm handling had timeouts.